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FOREWORD 


This  report  is  an  outgrowth  of  the  work  of  the 
Defense  Science  Board  Task  Force  on  Test  and  Evaluation, 
and  the  checklists  herein  have  been  derived  from  the 
study  of  past  major  weapon  system  programs. 

The  T&E  expert  in  reading  this  volume  will  find 
many  precepts  which  will  strike  him  as  being  too  obvious 
to  be  included  in  checklists  of  this  type.  These  items 
are  included  because  examples  were  found  where  even  the 
obvious  has  been  neglected,  not  because  of  incompetence 
or  lack  of  personal  dedication  by  the  people  in  charge 
of  the  program,  but  because  of  financial  and  temporal 
pressures  which  forced  competent  managers  to  compromise 
on  their  principles.  It  is  hoped  that  the  inclusion  of 
the  obvious  will  prevent  repetition  of  the  serious  errors 
which  have  been  made  in  the  past  when  such  political, 
economic  and  temporal  pressures  have  forced  project 
managers  to  depart  from  the  rules  of  sound  engineering 
practices. 

In  the  long  run,  taking  short  cuts  during  T&E  to 
save  time  and  money  will  result  in  significant  increases 
in  the  overall  costs  of  the  programs  and  in  the  delay  of 
the  delivery  of  the  corresponding  weapon  systems  to  the 
combatant  forces. 


T&E  GUIDELINES  FOR  COMMAND  AND  CONTROL  SYSTEMS 


The  checklist  items  presented  here  are  specifically  applicable  to 
command  and  control  testing  and  evaluation.  It  is  suggested  that  the  user 
of  this  volume  also  refer  to  the  Report  of  the  Defense  Science  Board  on 
Test  and  Evaluation  which  contains  general  checklist  items  also  applicable 
to  this  system  T&E  program.  The  checklist  items  presented  here  are  organ¬ 
ized  into  time  phases  of  the  acquisition  process  oriented  to  the  DSARC  cycle. 

The  checklists  cover  various  aspects  of  the  major  activities  that 
should  be  underway  during  a  given  time  period.  Hence,  a  checklist  might 
cover  the  (1)  evaluation  of  work  that  occurred  in  the  previous  phase, 

(2)  conduct  of  tests  planned  in  the  previous  phase  and  executed  in  the 
subject  phase,  and  (3)  plans  and  other  preparatory  actions  for  test  sche¬ 
dules  to  be  conducted  in  a  subsequent  phase.  For  reasons  such  as  this, 
items  on  some  subjects,  such  as  development  test  plans,  may  appear  in  more 
than  one  phase.  In  addition,  since  the  Services  and  the  DSARC  have  flexi¬ 
bility  in  deciding  how  rapidly  to  progress  in  the  validation  phase,  there 
may  be  cases  where  the  Request  for  Proposals  (RFPs),  proposal  evaluations, 
source  selections,  or  contract  negotiations  may  occur  after  the  DSARC 
approves  full-scale  development  instead  of  before.  For  this  reason,  it 
is  recommended  that  previous  checklists  in  the  Validation  Phase  be  re¬ 
viewed  when  entering  the  Full-Scale  Engineering  Development  Phase.  The 
following  are  the  phases  used  in  this  report. 

CONCEPTUAL  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  evaluating  T&E 
activities  during  the  Conceputal  Phase  of  the  acquisition  of  the  system. 

This  phase  (often  research  and  exploratory  development)  precedes  the  first 
DSARC  milestone  and  is  focused  on  the  development  of  a  system  concept  that 
offers  high  prospects  of  satisfying  an  identified  military  need. 

Although  not  called  for  in  DoD  Directive  5000.1  specifically,  the 
objective  of  this  phase  should  be: 
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1.  To  verify  that  there  is  a  military  need  for  the  proposed 
system, 

2.  To  demonstrate  that  there  is  a  sound  physical  basis  for  a  new 
weapon  system. 

3.  To  formulate  a  concept,  based  on  demonstrated  physical 
phenomena,  for  satisfying  the  military  need. 

4.  To  show  that  the  proposed  solution  is  superior  to  its  com¬ 
petitors  in  terms  of  potential  effectiveness,  probability  of 
success,  probable  cost,  impact  on  the  U.S.  military  posture, 
and  development  risks. 

5.  To  analyze  the  technology  outlook  and  the  military  need  to 
show  that  it  is  better  to  start  advanced  development  now 
rather  than  to  wait  for  future  technological  improvements. 

6.  To  identify  the  key  risk  areas  and  critical  issues  that  need 
to  be  resolved  before  full-scale  development  is  initiated. 

The  most  important  product  of  this  phase  is  the  Development  Concept 
Paper  (DCP)  or  its  equivalent.  The  DCP  defines  program  issues,  including 
special  logistics  problems,  program  objectives,  program  plans,  performance 
parameters,  areas  of  major  risk,  system  alternatives,  and  acquisition 
strategy. 


VALIDATION  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  conducting  T&E 
during  the  Validation  Phase  (the  time  between  when  the  DSARC  recommends 
approval  of  the  DCP  for  the  first  time  and  when  the  DSAPC  recommends  full- 
scale  development  of  the  system) . 

While  these  objectives  are  not  spelled  out  in  the  DoD  Directive  5000.1, 
the  objectives  of  the  Validation  Phase  should  be  to  confirm; 

1.  The  need  for  the  selected  system  in  consideration  of  the  threat, 
system  alternatives,  special  logistics  needs,  estimates  of 
development  costs,  preliminary  estimates  of  life  cycle  costs 
and  potential  benefits  in  context  with  overall  DoD  strategy  and 
fiscal  guidance. 

2.  The  validity  of  the  operational  concept. 

3.  That  development  risks  have  been  identified  and  solutions  are 
in  hand. 

4.  Realism  of  the  plan  for  full-scale  development. 
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In  the  pursuit  of  the  above  objectives,  it  is  likely  that  advanced 
development  T&E  will  be  conducted  to  resolve  issues.  In  some  cases,  an 
RFP  for  full-scale  engineering  development  will  be  prepared,  proposals 
will  be  received  and  evaluated,  and  contracts  negotiated  in  preparation 
for  seeking  DSARC  approval  for  the  next  phase.  Therefore,  some  checklist 
items  are  included  to  help  ensure  that  this  work  properly  reflects  the 
T&E  interests  in  this  and  subsequent  phases.  For  example,  the  RFP  must 
include  adequate  guidance  to  ensure  that  sufficient  resources  and  time  are 
available  so  that  engineering  effort  can  properly  support  the  initial  DT&E 
with  hardware,  software,  technical  data,  and  training. 

The  primary  emphasis  of  OSD/T&E  activities  is  with  items  3  and  4  above. 
Special  attention  should  be  given  to  the  planning  of  IOT&E  activity  as  it 
is  incorporated  in  the  engineering  development  contract  as  well  as  the 
DT&E  associated  with  addressing  the  critical  issues  and  areas  of  major 
risk  identified  in  the  DCP. 

FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in 
conducting  T&E  during  the  Full-Scale  Engineering  Development  Phase.  This 
includes  the  major  DT&E  and  the  IOT&E  conducted  prior  to  the  major  pro¬ 
duction  decision.  By  this  time,  the  C&C  system  is  well-defined  and  is 
becoming  a  unique  item  and,  hence,  sound  judgment  must  be  applied  in  using 
these  checklist  items. 

To  enter  the  Engineering  Development  Phase,  the  DSARC  will  have: 

•  Confirmed  the  need  in  consideration  of  the  threat,  alternatives, 
logistic  needs,  cost,  and  benefits. 

•  Identified  development  risks. 

•  Confirmed  the  realism  of  the  development  plan. 

Given  the  above,  the  primary  objectives  of  the  DT&E  should  be  to: 

1.  Demonstrate  that  the  engineering  and  design  and  development 
process  is  complete  and  that  the  design  risks  have  been  mini¬ 
mized  (the  system  is  ready  for  production). 

2.  Demonstrate  that  the  system  will  meet  specifications. 
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The  primary  objectives  of  the  IOT&E  should  be  to: 

3.  Assess  operational  suitability  and  effectiveness, 

4.  Validate  organizational  and  employment  concepts. 

5.  Determine  training  and  logistic  requirements. 

In  addition,  the  validity  of  the  plan  for  the  remainder  of  the  program 
must  be  confirmed  by  the  DSARC  before  substantial  production/deployment 
will  be  recommended  to  the  Secretary  of  Defense. 

The  level  of  OSD/T&E  activity  is  highest  during  this  phase.  The 
IOT&E  plan  must  be  designed,  the  tests  conducted,  and  the  data  analyzed 
to  evaluate  the  inputs  associated  with  the  primary  objectives.  These 
tests  should  not  be  conducted  until  the  primary  objectives  of  the  DT&E 
have  been  met.  Thus,  OSD/T&E  activity  is  required  to  assess  that  the  DT&E 
major  milestone — the  system  is  ready  for  production — has  been  achieved. 
Close  monitoring  of  the  T&E  Service  activity  is  required  during  the  latter 
stages  of  this  phase. 

SUBSTANTIAL  PRODUCT ION /DEPLOYMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in  con¬ 
ducting  T&E  after  the  substantial  production  decision  has  been  made  by  the 
DSARC.  This  includes  DT&E  and  follow-on  OT&E  to  be  conducted  on  the  early 
production  items. 

To  enter  the  Production/Deployment  Phase,  the  DSARC  will  have  re¬ 
viewed  the  program  to  confirm: 

•  The  need  for  the  system. 

•  A  practical  engineering  design  with  adequate  consideration  of 
production  and  logistic  problems  is  complete. 

*5  All  technical  uncertainties  have  been  resolved  and  opera¬ 
tional  suitability  has  been  determined  by  T&E. 

•  The  realism  of  the  pdan. 

The  primary  objective  of  the  DT&E  in  this  phase  should  be  to: 

1.  Verify  that  the  production  system  meets  specifications. 

The  primary  objectives  of  the  follow-on  OT&E  should  be  to: 
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2.  Validate  the  operational  suitability  and  effectiveness. 

3.  Optimize  organization  and  doctrine. 

4.  Validate  training  and  logistic  requirements. 

At  this  point,  the  OSD/T&E  activity  is  similar  to  that  in  the 
previous  phase;  however,  much  of  the  testing  is  verification  that  the 
production  system  performance  is  as  expected.  Hence,  most  of  the  items 
in  the  previous  phase  are  appropriate  to  this  phase,  especially  those 
related  to  OT&E. 
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I .  CONCEPTUAL  PHASE 


I .  CONCEPTUAL  PHASE 


The  prime  objective  of  the  Conceptual  Phase  of  a  Command  and  Control 
program  is  to  justify  the  program.  The  genesis  of  the  program  is  usually 
built  on  a  recognized  need  where  existing  Command  and  Control  systems 
have  serious  deficiencies,  cost  to  modify  them  adequately  would  be  high, 
and  a  more  efficient  new  system  is  proposed. 

The  new  concept  is  usually  based  on  studies,  ^IR&D  by  contractors, 
evidence  of  military  applications  of  commercial  gear,  and  other  exploratory 
and  advanced  development. 

The  test  and  evaluation  checklist  for  this  Conceptual  Phase  for 
Command  and  Control  systems  touches  on  the  following  subjects: 

1.  Conceptual  Test  Philosophy 

2.  The  Importance  of  Software  Testing 

3.  Software  Test  Scheduling  -  Contractors1  Facilities 

4.  Evaluation  of  Exploratory  Development  Tests 

5.  Feasibility  Testing  for  Field  Compilers 

6.  Application  Patches 

7.  Evaluation  of  Test  Plan  Scheduling 

8.  Type  Personnel  Needs  -  Effects  on  T&E 

9.  Planning  for  Joint  Service  OT&E  Before  DSARC  I 
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1.  CONCEPTUAL  TEST  PHILOSOPHY 


T&E  planners  must  understand  the  nature  of  Command  and  Control 

systems  early  in  the  Conceptual  Phase. 

In  a  complex  Command  and  Control  system  a  total  systems  concept  has 
to  be  developed  at  the  beginning.  Total  systems  life  cycle  must  be 
analyzed  so  the  necessary  requirement  for  the  design  can  be  established. 
Only  with  this  in  hand  can  targets  be  established  and/or  the  proper  trade¬ 
off  studies  made.  Whenever  two  major  systems  are  to  be  connected,  test 
and  evaluation  should  make  special  analysis  of  the  interface (s)  and  from 
the  systems  definition  phase  and  thereafter  should  test  the  credibility 
and  durability  of  the  interface (s)  and  monitor  change  in  the  interface (s) 
for  signs  of  instability  of  the  program(s). 

Central  control  of  interfaces  and  changes  to  inter-connected  systems 
is  also  required  since  systems  which  interface  usually  become  inter¬ 
dependent. 

Total  systems  concept  must  recognize  that  C&C  systems  will  evolve/ 
change  over  their  expected  life  span — due  to  changes  in  threat,  require¬ 
ment,  and  technology  and  that  this  may  dictate  need  for  flexibility  in 
hardware/software  approach.  Recognize  change  as  a  way  of  life  in  C&C 
systems  and  plan  for  it  and  test  for  ability  to  accept  change. 

The  variety  of  expected  users  must  be  understood  as  well  as  the 
expected  modifications  in  operational  environment  and  this  must  be 
extended  to  consider  prospective  changes  due  to  changes  in  threat, 
requirements  or  technology.  Remember  that  a  C&C  system  is  employed  for 
the  first  time  about  six  years  after  its  initiation  and  should  live  for 
at  least  six  years  thereafter.  Agreement  must  be  reached  between  the 
users  (whom  the  system  must  satisfy),  the  operators,  the  developers  and 
the  testers  as  to  what  the  system  is  intended  to  do,  and  as  to  the  way 
tests  can  predetermine  ultimate  performance. 

2.  THE  IMPORTANCE  OF  SOFTWARE  TESTING 

Testers  should  recognize  that  software  is  a  pacing  item  in  Command 

and  Control  systems  development. 
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The  pacing  item  in  most  (if  not  all)  automated  C&C  systems  proves 
to  be  the  software.  This  is  often  not  recognized  by  the  developer, 
tester,  or  the  user  who  are  traditionally  hardware-oriented.  The  result 
is  that  acceptance  of  the  developmental  system  from  the  contractor  is 
in  large  measure  based  on  compliance  with  hardware  specifications  fol¬ 
lowing  review  of  contractor-conducted  tests,  while  software  acceptance 
may  all  too  often  be  based  on  limited  and  selected  demonstrations  by 
the  contractor.  The  upshot  is  that  the  government  accepts  a  system  that 
has  not  been  tested  or  demonstrated  under  nfull  load"  conditions..  Having 
separate  hardware  and  software  contractors  creates  real  problems  as  to 
who  is  responsible  for  insuring  that  the  "system"  works — this  has 
plagued  C&C  systems  for  years.  It  is  best  to  have  one  contractor 
responsible  for  overall  system  integration.  If  more  than  one  contractor 
is  involved,  e.g. ,  separate  software  and  hardware  developers,  then  test 
plans  should  be  so  phased  as  to  require  successful  software  demonstration 
(e.g.,  by  well  conceived  emulation  or  translation) — followed  by  tests  to 
ensure  adequate  systems  integration — followed  by  total  system  test  and 
demonstration  prior  to  government  acceptance. 

3.  SOFTWARE  TEST  SCHEDULING  -  CONTRACTORS’  FACILITIES 

Provision  should  be  made  for  inclusion  of  software  T&E  during  each 

phase  of  C&C  systems’  acquisition.  Availability  of  contractors’ 

facilities  should  be  considered. 

Test  and  evaluation  should  insure  that  any  software  products  are 
tested  appropriately  at  the  completion  of  each  phase.  Software  has  often 
been  developed  more  as  an  add-on  than  as  an  integral  part  of  the  overall 
system.  Software  requirements  need  the  same  consideration  as  hardware 
requirements  in  the  Validation  Phase.  Usual  practices  often  do  not  suf¬ 
ficiently  provide  for  testing  the  software  subsystem  concept.  Often  the 
facilities  available  to  contractors  for  software  development  and  verifi¬ 
cation  are  critical  to  schedule  and  cost. 
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4.  EVALUATION  OF  EXPLORATORY  DEVELOPMENT  TESTS 


Care  should  be  exercised  in  evaluating  results  of  tests  conducted 

during  exploratory  development  of  Command  and  Control  Systems. 


Results  of  tests  conducted  during  exploratory  development  and  which 
most  likely  have  been  conducted  on  brassboard,  breadboard,  or  modified 
existing  hardware  should  be  evaluated  with  special  attention  to  items  such 
as: 


(a)  The  packaging  of  the  hardware  may  significantly  affect  the  per¬ 
formance  characteristics  so  that  the  suggested  proof  of  valida¬ 
tion  is  inconclusive.  A  case  in  point  is  the  experimental 
van-mounted,  relatively  immobile  Tactical  Operational  System 
(TOS)  for  Europe  versus  the  current  highly  mobile  TOS  test 

bed  configuration. 

(b)  The  laboratory  type  environment  in  which  the  hardware  was  tested 
may  preclude  the  generation  of  data  needed  to  validate  that  the 
concept  and  technology  approach  will  be  applicable  to  an  opera¬ 
tional  environment.  The  close  and  necessary  interaction  between 
the  Ultimate  user  and  the  developer  of  C&C  systems  emphasizes 
this  point. 

(c)  The  tests  may  not  include  signals  and  noise  sources  representa¬ 
tive  of  those  that  might  be  expected  in  an  operational  environ¬ 
ment.  Since  C&C  systems  depend  on  communication  links  this  point 
is  especially  important. 

(d)  Software  is  highly  hardware-dependent  and  in  most  cases  is  not 
transferable  from  one  system  to  another.  For  example,  software 
developed  for  the  Army's  initial  Tactical  Operations  Systems 
(TOS)  in  Europe  is  not  transferable  to  the  current  TOS  develop¬ 
ment  program.  Thus,  test  results  obtained  previously  have  some, 
but  limited,  value  to  the  present  program. 


5.  FEASIBILITY  TESTING  FOR  FIELD  COMPILERS 

Early  test  planning  should  allow  for  simulation  of  the  computer 

system  to  test  for  field  use  of  compilers,  where  applicable. 

If  the  development  plan  for  a  tactical  C&C  system  assumes  the  use  of 
a  compiler  in  the  field,  the  plan  should  provide  for  early  DT&E  simula¬ 
tion  of  the  computer  system  to  test  the  feasibility  of  using  the  compiler 
in  the  field.  Appropriate  simulations  or  analyses  done  early  will  allow 
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for  computer  changes  during  DT&E  that  could  not  be  accomplished  rapidly 
enough  to  avoid  disrupting  the  subsystems  and  system  IOT&E  schedule. 

6 .  APPLICATION  PATCHES 

Early  T&E  evaluation  of  procedures  for  application  patches  is 

needed . 

In  real  time  C&C  systems,  early  DT&E  verification  of  the  procedure 
proposed  for  application  patches  is  indispensable.  If  a  compiler  is  to 
be  used  in  the  field,  tests  must  be  made  to  ensure  that  enough  memory 

is  provided;  if  the  patches  are  to  be  provided  off  line  from  a  remote 
station,  the  particular  method  of  communication  necessary  to  specify  the 
patch  and  to  obtain  the  program  must  be  tested  as  well.  Procedures 
must  be  developed  and  tested  to  insure  accuracy  of  all  "transmitted" 
program  changes. 

7.  EVALUATION  OF  TEST  PLAN  SCHEDULING 

Milestones  should  be  event-oriented,  not  calendar-oriented. 

In  evaluating  the  adequacy  of  the  scheduling  as  given  by  test  plans, 
it  is  important  that  milestones  be  tied  to  the  major  events  of  the  C&C 
system  (meeting  stated  requirements)  and  not  the  calendar.  As  a  result, 
milestones  should  be  flexible  with  respect  to  time.  The  acquisition 
process  should  be  based  on  the  achievement  of  major  milestones  and  suf¬ 
ficient  time  and  resources  allowed  between  these  milestones.  -For  example, 
flexibility  must  not  be  hampered  by  the  contracting  mechanism.  Con¬ 
tractors  should  be  required  to  demonstrate  successful  accomplishment  of 
technical  milestones  before  proceeding  to  the  next  phase  of  development. 
Remember  software  testing  and  retesting  takes  time.  Allow  for  it. 

8.  TYPE  PERSONNEL  NEEDS  -  EFFECTS  ON  T&E 

A  mix  of  personnel  with  different  backgrounds  affecting  T&E  is 

required. 
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Developers,  testers,  evaluators,  operators,  and  users  have  quite 
different  backgrounds  and  needs  which  affect  the  T&E  of  the  C&C  system. 
Each  has  a  different  approach  which  has  merit  and  utility  at  almost  all 
points  in  the  T&E  program.  A  mix  of  these  types  are  needed  throughout 
the  program.  Early  in  the  program,  the  lead  emphasis  should  be  from 
the  developer/tester,  with  strong  user  input  shifting  to  the  evaluator 
and  finally  the  operator,  but  at  all  times  all  parties  and  their  needs 
should  be  coordinated. 

9.  PLANNING  FOR  JOINT  SERVICE  OT&E  BEFORE  DSARC  I 

Joint  Service  Operation  Test  and  Evaluation  should  be  considered 
for  Command  and  Control  systems. 

If  the  operational  concept  for  the  new  C&C  system  envisions  inter¬ 
operability  with  another  Service  system,  then  Joint  Service  Tests  should 
be  mandatory  and  should  be  planned.  An  analysis  of  the  impact  of  this 
type  of  testing  on  time  and  resources  needed  in  the  program  should  be 
conducted  before  DSARC  I. 
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II 


VALIDATION  PHASE 


II.  VALIDATION  PHASE 


The  major  concerns  in  this  phase  are  the  conducting  of  issue  tests 
and  critiquing  of  test  and  evaluation  plans  for  the  remainder  of  the 
program  and  particularly  in  the  Engineering  Development  and  IOT&E  phase. 


The 

checklist  includes : 

1. 

Test  Prototypes 

2. 

Test  Objectives  -  Critical  Issues 

3. 

Real  Time  Software  -  Demonstration 

of  Application  Patches” 

4. 

Independent  Software  Test-User  Group 

5. 

System  Error  Conditions 

6. 

Test  Schedule  -  Software  Debugging 

7. 

Real  Time  Operating  Systems 

8. 

Software  Test  Analysis 

9. 

System  Interfaces 

10. 

Human  Factors 

11. 

Degraded  Operations  Testing 

12. 

Test  Bed 

13. 

Software-Hardware  Interfaces 

14. 

Reproducible  Tests 

15. 

Cost-Effectiveness 

The 

reader  should  review  the  checklist 

items  in  the  previous  phase 

since  many  of  them  will  be  appropriate  for  this  phase. 
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1.  TEST  PROTOTYPES 


In  Command  and  Control  Systems,  prototypes  must  reasonably  resemble 

final  hardware  configuration  from  a  functional  use  standpoint. 

When  high  technical  risk  is  present,  development  should  be  structured 
around  the  use  of  one  or  more  test  prototypes  designed  to  prove  the 
system  concept  under  realistic  operational  conditions  before  proceeding 
to  engineering  development.  It  is  good  to  take  a  risk;  however,  when  an 
implied  commitment  to  production  is  involved  the  technology  should  be 
operationally  proof  tested  prior  to  commencing  full-scale  development. 

Avoid  the  temptation  of  thinking  that  anything  is  "state-of-the-art" 
until  it  is  working  in  the  field.  In  C&C  systems,  the  key  points  to 
remember  are  that  the  prototype  must  reasonably  resemble  final  hardware 
configuration  from  a  functional  use  standpoint  and  one  must  assemble 
a  reasonable  replica  of  the  total  system  so  that  the  functions  that  must 
be  performed  by  the  C&C  system  can  be  tested.  Often  a  representative 
slice  of  a  total  system  may  suffice  for  the  prototype  if  that  slice  may 
be  expected  to  perform  all  or  most  of  the  functions  of  the  total  system. 

2.  TEST  OBJECTIVES  -  CRITICAL  ISSUES 

In  addition  to  addressing  critical  technical  issues,  T&E  objectives 

during  the  Validation  Phase  should  address  the  functional  issues  of 

a  Command  and  Control  system. 

It  is  important  to  ensure  that  the  technical  and  operational  objectives 
of  the  tests  to  be  conducted  during  the  time  period  from  DSARC  I  to  DSARC 
II  address  the  major  critical  issues,  especially  technological  issues. 

In  addition  to  technical  issues,  the  functions  of  the  C&C  system  must  be 
clearly  defined  and  established  through  T&E  during  the  Validation  Phase. 
Each  test  should  have  a  single  objective  if  possible  and  the  objective 
should  be  simply  stated.  A  plan  for  the  conduct  of  the  test  and  the  data 
collection,  reduction,  and  analyses  must  be  in  sufficient  detail  so  that 
one  can  readily  evaluate  the  performance  of  the  system  and  whether  or  not 
the  test  objective  can  be  met.  A  relationship  between  the  identified 
performance  parameters  and  the  test  results  should  be  established  prior 
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to  the  conduct  of  the  test.  Further,  the  set  of  objectives  for  each  of 
the  tests  should  be  clearly  related  to  the  program  objectives.  When  this 
relationship  is  not  clear,  amplifying  data  should  be  required.  By  the  end 
of  the  systems  definition  phase,  test  and  evaluation  planners  should  make 
certain  that  test  criteria  are  established  so  there  is  no  question  as  to 
what  constitutes  a  test  and  what  performance  is  attained.  These  should 
be  agreed  to  by  the  developer,  the  tester  and  the  user. 

For  example,  if  the  program  objective  is  to  validate  the  concept  that 
the  proposed  C&C  system  will  result  in  providing  the  user  with  faster, 
more  reliable  and  complete  information  upon  which  to  base  a  decision,  then 
test  objectives  should  be  designed  to  measure  these  parameters  over  a 
variety  of  scenarios  and  to  evaluate  the  impact  of  these  parameters  on  the 
commander’s  decision  process. 

3.  REAL  TIME  SOFTWARE  -  DEMONSTRATION  OF  "APPLICATION  PATCHES” 

Tests  of  real  time  Command  and  Control  systems  should  include  demon¬ 

strations  of  interfaces  whereby  locally  generated  application  patches 

are  brought  into  being. 

Real  time  software  must  be  designed  to  permit  locally  generated  "appli- 
cation  patches"  to  cope  with  peculiar  local  situations  or  development  of 
new  procedures  or  threats,  while  at  the  same  time  maintaining  the  integrity 
of  standardized  software  or  interfaces.  Tests  of  all  real  time  systems 
should  include  demonstrations  of  methods  whereby  such  patches  are  brought 
into  being  within  an  interval  of  a  few  days. 

4.  INDEPENDENT  SOFTWARE  TEST-USER  GROUP 

An  independent  test-user  software  group  is  needed  during  early  soft¬ 

ware  qualification  testing. 

During  early  software  qualification  testing,  the  tests  themselves 
should  be  prepared  and  executed  under  the  direction  of  an  independent 
test-user  group  rather  than  under  the  control  of  original  programmers 
who  are  more  concerned  with  completing  the  software  system  development 
on  schedule  (with  minimal  fault  free  testing) . 
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5.  SYSTEM  ERROR  CONDITIONS 


A  set  of  tests  needs  to  be  designed  specifically  to  find  and  verify 

system  error  conditions. 

During  early  tests  of  software  programs,  a  set  of  tests  should  be 
designed  specifically  to  find  and  verify  error  conditions.  If  the  system 
is  designed  to  print  out  a  warning  message  when  incorrect  data  is  fed  in, 
then  a  series  of  tests  should  be  run  inputting  faulty  data  to  verify  cor¬ 
rect  performance. 

In  the  debugging  process,  a  variety  of  tests  should  be  run  to  verify 
algorithm  accuracy  and  to  assess  operator/user  impact  on  functional  appli¬ 
cations  programs,  e.g.,  formats,  operator  error.  Each  program  should  then 
be  rerun  until  it  executes  perfectly.  This  is  particularly  significant  in 
light  of  a  tendency  during  testing  to  carry  the  errors  forward  to  the 
next  higher  level  of  testing.  This  latter  approach  tends  to  permit  an 
excessive  number  of  errors  to  be  carried  forward  into  the  more  difficult 
test  phases. 

6.  TEST  SCHEDULE  -  SOFTWARE  DEBUGGING 

At  the  end  of  each  level  of  software  qualification  testing,  a  schedule 

provision  must  be  made  for  incorporating  corrections. 

In  C&C  systems  it  must  be  recognized  that  software  debugging  is  time 
consuming  and  will  require  tests  and  retests  during  the  T&E  cycle.  Soft¬ 
ware  testing  is  continuous  in  that  new  functions  in  all  likelihood  will  be 
added  in  the  T&E  cycle,  which  again  should  be  planned  for  in  the  T&E 
schedule.  For  example,  at  the  end  of  each  level  of  software  qualification 
testing,  a  schedule  provision  must  be  made  for  incorporating  corrections 
and  modifying  the  overall  system.  The  development  of  complex  software 
systems  is  simply  not  sufficiently  formalized  (as  is  hardware)  to  permit 
a  smooth  continuous  development  from  concept  to  implementation.  Error 
correction  and  design  modifications  will  be  required  after  each  level  of 
testing,  so  schedule  provisions  should  be  made  in  advance.  Fixing  soft¬ 
ware  bugs  needs  more  time  than  flow  charting  and  coding.  Schedules  that 
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do  not  take  this  into  account  should  be  rejected  unless  good  technical 
reasons  are  given. 


7.  REAL  TIME  OPERATING  SYSTEMS 

Test  plans  should  recognize  the  difference  between  an  operating  system 

and  a  real  time  operating  system  and  be  designed  accordingly. 


When  dealing  with  "real  time"  situations  in  Command  and  Control,  weapon 
assignment,  air  traffic  control,  and  the  like,  one  should  always  remember 
that  there  is  a  fundamental  difference  between  OS  (operating  systems)  and 
RTOS  (real  time  operating  systems) .  The  use  of  OS  where  RTOS  is  called 
for  is  usually  disastrous.  This  applies  with  particular  emphasis  to  the 
high  level  language  and  the  related  compiler  when  these  are  not  optimized 
for  minimum  execution  times. 

Test  plans  for  RTOS  software  should  be  designed  to: 

(a)  simulate  the  expected  maximum  environment  from  the  point 
of  view  of  traffic,  interference,  spurious  targets. 

(b)  Explore  simulated  displays  if  the  final  displays  are  not 
available. 

(c)  Employ  operators  who  are  not  so  skilled  as  to  make  the 
test  invalid  for  its  future  operational  employment. 

(d)  Test  for  continuity  of  operations  if  elements  of  the 
system  are  lost.  A  real  time  requirement  implies  a 
need  for  extremely  high  reliability,  usually  achievable 
only  through  redundant  on-line  equipment . 

(e)  Provide  for  on-line  (background)  diagnostic  routines. 

(f)  Provide  for  automatic  cutover  to  redundant  equipment 
or  some  other  form  of  graceful  degradation. 

(g)  Provide  for  user  notification  of  faults. 

8.  SOFTWARE  TEST  ANALYSIS 

Emphasis  should  be  placed  on  analyzing  a  reasonable  set  of  software 

tests  rather  than  on  generating  a  large  number  of  tests. 


In  developing  software  for  Command  and  Control  systems,  it  is  easier 
to  generate  tests  than  it  is  to  analyze  their  results.  A  reasonable  set 
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of  tests,  carefully  constructed,  analyzed,  and  documented  often  proves 
more  useful  than  a  much  larger  set  of  tests  performed  and  analyzed  less 
carefully.  Further,  sufficient  and  qualified  government  and/or  contractor 
resources  must  be  made  available  to  perform  the  required  analyses. 

Therefore,  planners  must  ensure  that  test  plans  call  for  a  carefully 
paced  software  test  program  with  emphasis  on  analyses  of  results  and  that 
adequate  resources  are  applied  to  the  job. 

9.  SYSTEM  INTERFACES 

Critical  attention  should  be  devoted  to  testing  interfaces  with  other 

C&C  systems  and  to  interfaces  between  subsystems. 

In  every  system  but  especially  in  C&C  systems,  DT&E  and  OT&E  should 
not  only  test  subsystems  and  systems,  but  interfaces  as  well.  Particular 
attention  should  be  devoted  to  interfaces  with  other  C&C  systems  and  to 

t 

the  interfaces  between  sensors  (e.g.,  radars) ,  communications  systems 
(e.g.,  modems) ,  and  the  specific  processors  (e.g.,  CPU).  Interface  with 
information  processing  C&C  systems  must  also  address  data  element  and 

i 

code  standardization  problems  if  data  is  to  be  processed  on-line. 

10.  HUMAN  FACTORS 

In  a  C&C  system  human  factors  must  be  considered  from  the  earliest 

prototype  designs  and  testing  provided. 

In  evaluating  Command  and  Control  systems,  human  factors  should  play 
a  large  role  and  must  be  considered  from  the  earliest  prototype  designs. 
Testing  should  be  conducted  to  determine  the  most  efficient  arrangement 
of  equipment  from  the  human  factor  standpoint,  e.g.,  displays  should  be 
arranged  so  as  to  be  viewed  from  an  optimum  angle  whenever  possible; 
adequate  maneuvering  room  within  the  installation  constraints  should  be 
allowed  considering  the  number  of  personnel  normally  manning  the  facility; 
and  console-mounted , controls  should  be  so  designed  and  located  as  to 
facilitate  operation,  minimize  fatigue  and  avoid  confusion.  Operators 
should  be  able  to  function  in  a  reasonably  comfortable  environment  (e.g.. 
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consider  lighting  levels,  temperature,  humidity  and  air  exchange  controls, 
chair  comfort  and  noise  levels). 

11.  DEGRADED  OPERATIONS  TESTING 

When  the  expected  operational  environment  of  a  C&C  system  suggests 

that  the  system  may  be  operated  under  less  than  finely  tuned  condi¬ 

tions,  tests  should  be  designed  to  allow  for  performance  measurements 

under  degraded  conditions. 

The  system  concept  and  possible  implementations  must  not  hinge  on 
the  requirements  for  the  system  or  subsystems  to  be  finely  tuned  when  the 
expected  operational  environment  suggests  that  this  will  not  be  likely. 

The  system  should  not  degrade  significantly  as  a  result  of  detuning  caused 
from  expected  operational  usage.  For  example,  if  the  capability  to  re¬ 
ceive  data  at  the  processing  center  is  expected  to  degrade  with  operational 
use  (and  the  effectiveness  of  the  system  depends  on  the  quality  and  com¬ 
pleteness  of  the  data) ,  then  tests  of  the  data  transmission  .capability 
should  be  conducted  under  varying  conditions  of  signal-to-noise  ratios 
to  establish  sensitivity  factors  and  under  varying  traffic  load  condi¬ 
tions  to  establish  saturation  factors.  The  developer  and  the  user  should 
agree  on  the  specification  of  the  acceptable  performance  envelope. 

12.  TEST  BED 

The  use  of  a  test  bed  for  study  and  experimentation  with  new  C&C 

systems  is  needed  early  in  the  Validation  Phase. 

Command  and  Control  systems  need  a  test  bed  starting  in  the  early 
validation  phase  for  study  and  experimentation  with  user  input.  This 
test  bed  is  required  through  the  full  period  of  the  system  design  and 
for  continued  program  maintenance  after  the  system  is  fielded.  The  test 
bed  requires  a  computer  with  software  that  works  and  that  the  user  can 
understand,  and  peripherals  that  are  truly  an  aid  to  the  user.  Testing 
early  versions  of  such  systems  with  user  participation  in  the  human 
acceptability  and  compatibility  environment  is  important.  Test  beds 
should  be  used  especially  in  the  validation  phase  of  systems  acquisition 
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prior  to  the  large  scale  commitment  of  resources.  Tests  should  be  per¬ 
formed  in  all  typical  operational  environments  to:  assess  the  need  for 
materiel  adjustments;  provide  hard  data  for  cost-effective  analyses; 
gain  user  acceptance;  assess  the  value  of  reduced  reaction  time  to  a 
variety  of  situations;  refine  software  procedures  and  applications;  refine 
user  requirements,  and  judge  the  impact  of  the  system  on  force  structure 
and  doctrine. 

13.  SOFTWARE-HARDWARE  INTERFACES 

The  software-hardware  interfaces  with  all  operational  back-up  modes 

to  a  new  C&C  system  should  be  tested  early  in  the  program. 

Software-hardware  interfaces,  particualrly  in  the  Command  and  Control 
area,  should  be  included  as  early  as  practical  in  all  operational  modes. 
Sometimes  the  operational  back-up  modes  do  not  get  tested  until  many  years 
after  the  system  is  operational.  On  occasion,  surprises  have  been 
experienced  when  these  modes  were  finally  tested. 

14.  REPRODUCIBLE  TESTS 

Test  plans  should  contain  a  method  for  allowing  full-load  message 

inputs  while  maintaining  reproducible  test  conditions. 

One  of  the  elements  associated  with  conducting  "full-load"  tests  on 
automated  systems  early  in  the  testing  cycle  is  the  importance  of  main¬ 
taining  a  reproducible  test.  It  is  extremely  difficult  (if  not  virtually 
impossible)  to  manually  generate  a  heavy  load  for  a  complex  C&C  system 
without  an  unreasonably  large  test  staff  and  even  then,  the  element  of 
human  error  is  present.  Therefore,  test  plans  should  contain  a  method 
(e.g.,  tape  recorder  message  input)  for  allowing  full-load  message  inputs 
while  maintaining  reproducible  test  conditions. 

15 .  COST-EFFECTIVENESS 

Field  test  data  is  needed  during  the  Validation  Phase  for  input  to 

cost  effectiveness  analyses  of  C&C  systems. 
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One  of  the  greatest  difficulties  associated  with  proposed  automated 
C&C  systems  is  to  demonstrate  that  they  are  cost  effective.  Unlike 
weapon  systems  that  can  graphically  show  their  potential  lethality  or 
destructiveness  (Pk)  on  the  battlefield,  the  value  of  automation  in  terms 
of  measures  of  effectiveness  is  a  difficult  and  elusive  problem.  It  is 
often  not  sufficient  to  show  that  the  automated  system  can  do  the  job 
"faster"  than  the  manual  system  or  that  more  information  can  be  made 
available  to  the  decision-maker  through  data  processing.  Paper  studies 
may  provide  some  answers  but  comparative  hard  field  data  based  on  realistic 
scenario-type  testing  is  indispensible .  Therefore  in  testing  new  tactical 
automated  Command  and  Control  systems  during  both  the  Validation  and 
Engineering  Development  Phases,  all  typical  battlefield  scenarios  in  which 
the  system  is  expected  to  operate  must  be  devised  and  comparative  side-by- 
side  tests  with  the  manual  or  older  system  conducted.  Sufficient  repeti¬ 
tions  with  change  of  "players"  must  be  allowed  to  provide  for  an  adequate 
data  base  and  the  human  factors  influence  on  the  test  results. 
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III.  FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 


III.  FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 


In  this  phase,  the  T&E  plans  developed  in  the  Validation  Phase  will 
be  refined,  and  the  development  testing  will  be  conducted.  IOT&E  plans 
will  similarly  be  refined;  personnel  will  be  assigned  and  trained;  and 
finally  the  testing  will  be  conducted. 

The  checklist  includes: 

1.  Operational  Testing 

2.  Acquisition  Strategy 

3.  Problem  Indications 

4.  Impact  of  Software  Failures 

5.  Critical  Issues 

6.  Displays 

7.  Pilot  Test 

8.  Publications  and  Manuals 

9 .  Power  Sources 

10.  OT&E  Reliability  Data 

11.  Subsystem  Tests 

12.  Communications 

13.  Maintenance 

14.  Continuity  of  Operations 

15.  Imitative  Deception 

16.  Demonstration  of  Procedures 

17.  Government  Furnished  Equipment  and  Facilities 

18.  User  Participation  in  T&E 

The  reader  should  review  the  checklist  items  in  the  previous  phases 
since  many  of  them  will  be  appropriate  for  this  phase. 
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1.  OPERATIONAL  TESTING 


Operational  testing  may  be  expensive  and  time  consuming  but  for 

C&C  systems  it  is  essential. 


Operational  testing  of  C&C  systems  is  essential.  It  is  also  expensive 
and  time  consuming,  but  the  value  received  is  worth  its  weight  in  not- 
delivered  systems.  Think  in  terms  of: 

(a)  Involving  operational  groups  in  test  planning  and  in 
establishing  measures  of  effectiveness,  so  that  the 
outcome  of  the  tests  will  be  accepted  as  being  opera¬ 
tionally  significant.  Remember,  a  C&C  system  will  not 
be  used  if  it  proves  to  be  a  burden  to  the  user. 

(b)  Determining  whether  the  scope  of  the  planned  tests  will 
provide  sufficient  data  to  justify  changes  in  the  eyes 
of  potential  users.  C&C  systems  are  so  closely  tied  to 
the  needs  of  the  user  that  changes  to  accommodate  his 
needs  must  be  expected. 

(c)  Comparing  the  scope  of  proposed  tests  against  checklists 
of  issues  frequently  raised  at  major  decision  milestones, 
to  assure  that  the  data  needed  for  such  decisions  will  be 
forthcoming  to  the  extent  this  is  possible  from  testing 
alone.  For  example,  a  major  issue  raised  in  C&C  systems 
is  their  cost-effectiveness.  Proposed  tests  must  include 
provisions  whereby  data  are  collected  to  assist  in  determin¬ 
ing  cost-effectiveness. 

(d)  Recognizing  in  the  formulation  of  test  plans  that  major 
system  decisions  are  judgments  based  on  a  wide  range  of 
qualitative  considerations,  rather  than  on  statistical 
compilations,  and  that  the  outcome  and  limitations  of 
operational  test  must  be  comprehensive  and  meaningful  to 
the  decision  makers  as  well  as  to  the  testing  community. 


2.  ACQUISITION  STRATEGY 

The  acquisition  strategy  for  the  system  should: 

(a)  Allow  for  a  sufficient  time  between  the  planned  end  of 
demonstration  testing  and  major  procurement  (as  opposed 
to  limited  procurement)  decisions  so  that  there  is  a 
flexibility  for  modification  of  plans  which  may  be  re¬ 
quired  during  the  test  phases  of  the  program.  For 
instance,  because  insufficient  time  was  allowed  for 
testing  one  recent  C&C  system,  the  program  and  the  contract 
had  to  be  modified  and  renegotiated. 
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(b)  Be  evaluated  relative  to  constraints  imposed  by: 

•  The  level  of  system  testing  at  various  stages  of  the 
RDT&E  cycle- 

•  The  number  of  test  items  available  and  the  schedule 
interface  with  other  systems  needed  in  the  tests,  such 
as  aircraft,  radars,  COMSEC  equipment,  communications 
gear,  power  generators*  etc, 

•  Support  required  to  assist  in  the  preparation,  conduct 
of  the  tests  and  the  analysis  of  test  results. 

(c)  Ensure  that  sufficient  dollars  are  available  not  only  to 
conduct  the  planned  T&E  but  to  allow  for  the  additional 
T&E  which  is  always  required  due  to  failures,  design 
changes,  etc. 


3.  PROBLEM  INDICATIONS 

It  is  important  to  establish  an  early  detection  scheme  for  manage¬ 

ment  to  determine  when  a  program  is  becoming  ill. 


Establish  an  early  detection  scheme  for  top  government  and  contractor 
management  to  determine  that  a  program  is  becoming  ill.  At  this  time 
there  may  be  a  good  possibility  of  recovery.  Some  of  the  indications  of 
trouble  are: 

•  The  number  of  software  deficiencies  is  excessive  and  fixes 
are  not  readily  identifiable. 

•  A  major  component  failure  such  as  the  central  processor. 

•  Any  repetitive  failure  of  a  key  component  or  a  component 
which  is  common  throughout  the  system  such  as  power  supplies. 

•  A  revision  of  schedule  or  incremental  funding  that  exceeds 
the  original  plan.  Predicted  downstream  recovery  may  not 
have  a  realistic  basis. 

4.  IMPACT  OF  SOFTWARE  FAILURES 

Prior  to  any  production  release,  the  impact  of  software  failures 

on  overall  system  performance  parameters  must  be  considered. 


By  the  end  of  the  phase,  the  impact  of  software  errors  and  failures 
on  overall  system  availability,  mean  time  between  failure  (MTBF)  and 
mean  time  to  repair  (MTTR)  must  be  known.  Testers  normally  compute  these 
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performance  parameters  based  on  hardware  test  data  only  and  software 
problems  are  treated  as  a  separate  area.  In  early  testing  this  may  make 
sense  from  a  data  management  problem  but  before  any  commitment  to  system 
production  is  made,  hardware  and  software  results  must  be  integrated  to 
determine  overall  system  performance  and  suitability.  As  an  example, 
past  experience  with  both  commercial  and  military  ADP  systems  has  shown 
that  system  downtime  due  to  software  problems  has  exceeded  that  of  hard¬ 
ware  by  at  least  three  to  one,  and  that  is  a  conservative  figure. 


5.  CRITICAL  ISSUES 

IOT&E  should  provide  the  answers  to  some  critical  issues  peculiar 

to  C&C  systems. 


Some  of  the  critical  issues  that  OT&E  of  Command  and  Control  systems 
should  answer  are: 

•  Is  system  mission  reaction  time  a  significant  improvement 
over  present  systems? 

•  Is  a  back-up  mode  provided  for  use  when  either  airborne 
or  ground  system  exhibits  a  failure? 

•  Can  the  system  be  transported  as  operationally  required 

by  organic  transport?  (Consider  ground,  air  and  amphibious 
requirements) . 

•  Is  there  a  special  requirement  for  site  preparation?  (For 
example,  survey,  antenna  siting). 

•  Can  the  system  be  erected  and  dismantled  in  times  specified? 

Are  these  times  realistic? 

•  Does  relocation  affect  system  alignment? 

•  Does  system  provide  for  operation  during  maintenance? 

•  Can  maintenance  be  performed  on  site  on  non-shelterized 
exposed  subsystems  during  adverse  weather  conditions, 
e.g. ,  radars? 

6.  DISPLAYS 

The  display  subsystems  of  a  C&C  system  should  provide  an  essential 

function  to  the  user. 


26 


Displays  are  key  subsystems  of  a  Command  and  Control  system.  They 
provide  the  link  that  couples  the  operator  to  the  rest  of  the  system  and 
are  therefore  often  critical  to  its  success.  In  testing  displays  and 
their  associated  consoles,  the  following  factors  should  be  considered: 

(a)  Does  the  display  provide  for  the  selection  of  a  variety  of 
relevant  items  of  information  under  user  control  in  order  to 
avoid  viewer  saturation,  e.g.,  overlay? 

(b)  Is  the  information  displayed  organized  to  ensure  the  asso¬ 
ciation  of  related  items  and  optimum  viewing? 

(c)  Is  the  data  presented  at  a  rate  commensurate  with  the  nature 
of  the  problem,  e.g.,  many  tactical  applications  require  only 
relatively  static  displays? 

(d)  Ambient  illumination,  display  size,  and  size  of  the  materiel 
displayed . 

(e)  Physical  layout. 

(f)  IOT&E  should  determine  whether  the  display  provides  an  essen¬ 
tial  (and  cost  effective)  function  for  the  user  or  is  it  in 
fact  nice  to  have  but  its  function  is  better  performed,  for 
examp le ,  manual ly . 

7.  PILOT  TEST 

A  pilot  test  should  be  conducted  prior  to  IOT&E  so  that  sufficient 

time  is  available  to  make  the  necessary  changes  to  the  IOT&E  as 

dictated  by  the  results  of  the  pilot  test. 


Before  any  operational  test  for  demonstration  of  operational  suita¬ 
bility  and  effectiveness  is  conducted,  a  pilot  test  should  be  held  with 
the  primary  purpose  of  shaking  down  the  test  plan,  and  the  data  analysis 
plan.  A  secondary,  but  vital  purpose  should  be  to  provide  final  training 
for  the  test  participants.  The  pilot  test  should  be  conducted  sufficiently 
prior  to  the  OT&E  so  that  sufficient  time  is  available  to  make  the  neces¬ 
sary  changes  to  the  OT&E  as  dictated  by  the  results  of  the  pilot  test. 

For  example,  pilot  tests  for  C&C  systems  may  be  conducted  with  reduced 
communications  ranges,  with  minimal  operator  manning  and  with  "canned11 
message  inputs . 


27 


8.  PUBLICATIONS  AND  MANUALS 


It  is  imperative  that  all  system  publications  and  manuals  be  completed, 

reviewed  and  selectively  tested  under  operational  conditions  prior  to 

the  beginning  of  overall  system  suitability  testing. 

Equipment  and  software  publications,  e.g.,  operating  and  maintenance 
manuals  are  often  incomplete  and  late  relative  to  DT&E  and  IOT&E.  In  a 
Command  and  Control  system  that  incorporates  many  separate  and  sometimes 
diverse  pieces  of  hardware,  the  effects  of  this  have  serious  impact  on  the 
overall  schedule  and  effectiveness  of  the  testing.  For  example,  maintenance 
manuals  that  are  disorganized  and  incomplete  hinder  expeditious  trouble¬ 
shooting  and  interpretation  of  malfunction  indications.  Therefore,  it  is 
imperative  that  all  equipment  and  software  publications  be  completed,  tho¬ 
roughly  reviewed  by  the  users  and  selectively  field  tested  by  appropriate 
military  users  prior  to  the  beginning  of  overall  system  suitability  testing. 

9.  POWER  SOURCES 

Mobile  prime  power  sources  are  usually  provided  as  GFE  and  can  be  a 

problem  area  in  testing  C&C  systems. 

Essentially  all  field  installations  of  Command  and  Control  systems 
require  mobile  prime  power  sources  which  are  usually  provided  as  GFE. 

These  equipments  should  be  evaluated  using  at  least  the  following  character¬ 
istics: 

•  Frequency  and  voltage  stability  sufficient  to  meet  C&C 

system  requirements. 

•  Weight  and  volume 

•  Logistical  support  required 

•  Fuel 

-  Availability  in  field 

-  Handling  difficulties 

-  Weight 

•  Efficiency 

•  Noise  level 

•  Ability  to  operate  in  all  environments. 
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•  Reliability 

•  Maintenance 

Generator  outputs  should  be  tested  under  varying  load  conditions 
imposed  by  the  supported  system  for  evidence  of  surge  voltages,  spikes 
or  other  transients  that  could  affect  the  power  distribution  system  of 
the  Command  and  Control  facility.  The  system  should  also  be  tested  with 
short  term  generator  outages,  i.e.,  a  minute  or  so.  For  example,  can  the 
system  perform  during  a  short  term  generator  outage  using  battery  power? 

10.  IOT&E  RELIABILITY  DATA 

IOT&E  can  provide  valuable  data  on  the  operational  reliability  of 

a  C&C  system  which  cannot  be  obtained  through  DT&E. 

Factors  such  as  operator  error  failures,  e.g.,  using  incorrect  mes¬ 
sage  formats,  and  environmentally-induced  failures  should  be  looked  for  in 
the  operational  tests  and  investigated  to  determine  if  serious  problems 
are  underlying  reasons  for  the  failures.  Especially  important  is  the 
procedure  used  to  evaluate  the  operational  reliability  of  the  C&C  system 
as  determined  by  the  relatively  small  amount  of,  but  significant,  data 
obtained  through  IOT&E  and  the  larger  amounts  of  data  on  hardware  and 
software  design  reliability  collected  through  DT&E. 

11.  SUBSYSTEM  TESTS 

Every  major  subsystem  of  a  C&C  system  should  have  a  successful  DT&E 

prior  to  beginning  of  overall  system  operational  testing. 

Operational  tests  designed  to  demonstrate  and  assess  the  operational 
suitability  and  effectiveness  of  the  C&C  system  should  not  be  conducted 
unless  every  major  subsystem,  critical  to  the  primary  mission  has  had  a 
successful  DT&E  and  has  been  incorporated  into  the  system  as  intended  in 
service  use.  If,  for  whatever  reasons  this  rule  is  not  followed,  then  at 
a  minimum  it  is  imperative  that  a  substitute  subsystem  be  used  which 
interfaces  with  the  other  subsystems  in  such  a  way  as  to  not  significantly 
affect  their  performance.  However,  the  decision  to  let  a  major  production 
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contract  on  the  system  should  not  be  made  until  the  entire  system  has 
been  demonstrated  to  be  operationally  acceptable  to  the  user. 

12.  COMMUNICATIONS 

C&C  systems  must  be  tested  in  the  appropriate  electromagnetic 

environment  to  determine  performance  of  its  communications  system. 

Communication  systems  remain  a  prerequisite  element  in  Command  and 
Control  systems.  Two  categories  of  communication  problems  are  involved. 

One  is  the  high  data  rate  computer-to-computer  interchange  of  information 
and  the  other  is  the  low  bandwidth  data  link  circuits  that  tie  message 
entry  devices  to  readout  devices  at  remote  locations.  The  former  category 
is  very  sensitive  to  such  considerations  as  circuit  noise,  delay  dis¬ 
tortion  and  interference  generated  either  intentionally  or  unintentionally. 
The  latter  is  sensitive  to  ECM,  local  propagation  conditions  and  mutual 
interference  problems. 

Therefore,  Command  and  Control  systems  must  be  designed  with  the 
hostile  combat  area  in  mind  and  tested,  first  with  electromagnetic 
analyses  and  then  in  an  environment  that  approaches  as  realistic  condi¬ 
tions  as  possible.  For  example,  if  the  Command  and  Control  system  under 
development  for  one  service  is  likely  to  be  used  at,  with,  or  near  facili¬ 
ties  of  another  service,  then  it  is  important  this  be  considered  in 
planning  for  the  test  scenarios. 

13.  MAINTENANCE 

In  IOT&E,  maintenance  testing  should  include: 

•  A  measurement  of  the  adequacy  of  the  maintenance  levels  and 
the  maintenance  practices; 

•  An  assessment  of  the  impact  that  the  maintenance  plan  has 
on  the  operational  reliability; 

•  The  accessibility  of  the  major  components  of  the  system  for 
field  maintenance,  e.g.,  are  cables  and  connectors  installed 
so  as  to  facilitate  access. 
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•  Verification  that  the  software  design  for  maintenance  and 
diagnostic  routines  and  procedures  are  adequate  and  that 
the  software  can  be  modified  to  accommodate  functional 
changes. 

14.  CONTINUITY  OF  OPERATIONS 

IOT&E  should  provide  for  an  impact  assessment  of  the  failure  of  any 

subsystem  element  of  a  C&C  system  on  overall  mission  effectiveness. 

Continuity  of  operations  is  an  essential  prerequisite  of  any  auto¬ 
mated  Command  and  Control  system  concept.  The  impact  on  mission  effective¬ 
ness  of  a  failure  of  any  subsystem  element,  especially  computer  centers, 
must  be  assessed.  Therefore,  IOT&E  should  be  designed  to  provide  the 
mechanism  whereby  the  assessment  can  be  made.  Tests  should  include 
employment  of  back-up  systems  and  ease  of  transition  from  one  system  to 
another  under  various  interrupt  conditions.  SOPs  for  accomplishing  the 
transition  should  be  evaluated. 

15.  IMITATIVE  DECEPTION 

IOT&E  should  provide  for  tests  to  assess  the  susceptibility  of  the 

data  links  of  a  C&C  system  to  imitative  deception. 

The  digital  data  links  associated  with  automated  Command  and  Control 
systems  are  likely  to  have  a  distinctive  signature  on  the  battlefield. 

The  possibility  of  enemy  use  of  imitative  deception  should  be  recognized. 

16.  DEMONSTRATION  OF  PROCEDURES 

Test  plans  should  include  a  procedural  demonstration  whereby  the 

tested  C&C  system  works  in  conjunction  with  other  systems. 

In  testing  C&C  systems,  it  is  imperative  that  the  test  plans  include 
a  demonstration  oj:  the  procedures  whereby  a  new  or  modified  system  works 
in  conjunction  with  other  systems  that  will  or  are  likely  to  coexist  in 
the  same  geographic  environment.  If  a  hardware,  software,  or  communica¬ 
tion  interface  is  required,  the  test  plans  must  include  these  interfaces. 


31 


17.  GOVERNMENT  FURNISHED  EQUIPMENT  AND  FACILITIES 


T&E  should  be  concerned  about  the  availability  of  GFE  equipment  as 

specified  in  the  proposed  contract. 


If  there  are  GFE  and  other  government  commitments  in  the  proposed 
contract,  be  concerned  about  the  following: 

(a)  Can  the  equipment  with  required  performance  be  available 
when  required?  Examples  are  COMSEC  equipment,  generators, 
communications,  shelters,  vehicles.  In  one  recent  C&C  test 
program,  the  required  generators  were  not  available  and  the 
COMSEC  equipment  design  required  some  redesign  of  the  C&C 
system  interface. 

(b)  Can  government  supported  facilities  provide  the  assistance 
required  at  the  time  needed?  If  not,  is  it  reasonable  to 
construct  the  required  facilities  (test  ranges,  airspace, 
firing  ranges,  instrumentation,  etc.)?  If  not,  what  alterna¬ 
tives  are  available? 

(c)  Avoid  contract  terms  on  fixed  price  contracts  that  vaguely 
commit  the  government.  Don't  include,  "government  support 
as  required"  or  "test  facilities  will  be  made  available 
when  needed". 


18.  USER  PARTICIPATION  IN  T&E 

The  varying  needs  of  the  user  for  a  C&C  system  make  it  mandatory 

that  he  participate  in  all  phases  of  T&E. 


It  is  imperative  that  the  user  participate  in  all  of  the  T&E  phases 
to  ensure  that  the  user  needs  are  represented  in  the  development  of  the 
system  concept  and  hardware/software.  Initially,  the  user  command  should 
play  an  advisor  role  during  the  feasibility  and  engineering  testing  pro¬ 
gram  and  gradually  assume  a  more  active  role  in  the  conduct  of  the  testing 
programs,  as  it  becomes  more  and  more  operational.  This  should  facili¬ 
tate  the  necessary  communication  and  interaction  between  the  developing 
and  user  command — especially  needed  during  the  DT&E  and  IOT&E  phases. 

This  is  especially  true  of  C&C  systems  where  the  varying  needs  of 
the  user  must  be  satisfied.  The  close  coupling  of  the  user  to  the  C&C 
system  necessitates  that  the  system  be  fully  matched  to  the  operator 
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personnel.  One  point  to  remember  during  OT&E  is  that  the  tests  must 
be  designed  to  assess  whether  the  C&C  system  will  be  a  burden  to  the  user. 
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IV.  SUBSTANTIAL  PRODUCTION/DEPLOYMENT  PHASE 


IV.  SUBSTANTIAL  PRODUCTION /DEPLOYMENT  PHASE 


This  phase  includes  follow-on  OT&E  that  is  performed  after  the  DSARC 
decision  for  substantial  production  and  deployment.  It  is  generally  con¬ 
ducted  with  production  hardware.  Prior  to  the  major  production  decision, 
the  DSARC  will  assess  the  adequacy  of  test  results  to  support  a  decision 
to  proceed  with  major  production  and  the  adequacy  of  plans  and  schedules 
for  any  remaining  testing. 

The  checklist  includes: 

1.  First  Article  Testing 

2.  Test  Planners  and  Evaluators 

The  reader  should  review  the  checklist  items  in  the  previous  phases, 
especially  Phase  III,  since  many  of  them  will  be  appropriate  to  this  phase. 


1. 


FIRST  ARTICLE  TESTING 


Conduct  first  article  testing. 

The  preproduction,  first  article,  testing  and  evaluation  should  be 
designed  and  conducted  to:  (1)  confirm  the  adequacy  of  the  equipment  to 
meet  specified  performance  requirements;  (2)  confirm  the  adequacy  of  the 
software  not  only  to  meet  current  user  needs  but  also  to  accommodate 
changing  needs;  and  (3)  determine  failure  modes  and  rates  of  the  total 
integrated  system.  This  activity  should  be  followed  by  FOT&E. 

2.  TEST  PLANNERS  AND  EVALUATORS 

Use  the  IOT&E  personnel  in  the  follow-on  OT&E  program. 

The  planners  and  evaluators  for  the  OT&E  of  the  production  system  can 
do  a  better  job  if  they  are  initially  involved  in  planning  and  conducting 
the  IOT&E.  The  program  plan  should  be  reviewed  to  ensure  that  the  FOT&E 
people  are  identified  for  IOT&E  participation  and  that  the  personnel  system 
of  their  service  retains  identity  of  these  people  for  use  in  planning,  con¬ 
ducting,  and  evaluating  FOT&E  which  may  not  be  run  until  a  year  or  two 
afterwards.  This  is  especially  critical  to  testing  of  C&C  systems  which 
require  a  reasonably  high  degree  of  training  and  knowledge  of  electronics, 
automatic  data  processing,  communications  and  especially  field  operating 
procedures,  tactics  and  doctrine. 
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